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Abstract 

In  many  government  weapon  systems,  sustaining  software  depends  heavily  on  organic 
engineering  efforts.  This  is  different  from  hardware  sustainment  (the  more  traditional  form  of 
sustainment),  which  often  depends  heavily  on  the  supply  chain  and  service  providers  and 
much  less  on  engineering  capability.  Because  of  this  shift,  a  larger  portion  of  sustainment 
funding  needs  to  be  allocated  to  improving  the  sustainment  infrastructure  within  government 
sustainment  organizations.  This  includes  the  engineering  processes,  tools,  and  skills  of 
engineering  staff.  Failure  to  recognize  this  need  in  a  timely  fashion  has  the  potential  to 
increase  sustainment  costs  and,  at  the  same  time,  degrade  system  performance. 

The  decisions  and  processes  are  complex  because  various  stakeholders  make  decisions  at 
different  times,  yet  these  decisions  are  interrelated,  impact  one  another,  and  create 
constraints  on  the  ability  of  the  sustainment  organization  to  fulfill  its  mission.  To  deal  with  the 
complexity  of  the  decision-making  process,  the  Software  Engineering  Institute  (SEI) 
developed  a  simulation  model  for  analyzing  the  effects  of  changes  in  demand  for  software 
sustainment  and  the  corresponding  funding  decisions.  The  model  allows  decision-makers  to 
analyze  multiple  allocation  strategies  in  response  to  changes  from  mission  command  and 
budget  authorities.  The  model  has  been  tested  and  calibrated  using  historical  data  and  is  now 
in  operational  use  by  the  Process  Resource  Team  at  the  Naval  Air  Weapons  Station  China 
Lake. 

Introduction 

The  ideas  for  this  model  were  developed  in  response  to  real-world  events.  One  DoD 
program  charged  with  sustaining  a  20+  year  old  aircraft  system  asked  the  SEI  how  it  could 
justify  the  capital  investment  necessary  to  update  its  test  and  support  systems  and  the 
supply  chain.  Some  of  the  parts  were  past  end-of-life,  making  them  difficult  to  source,  and 
the  radar — a  major  technology  component — had  been  updated  without  updating  the  test 
equipment.  The  situation  was  placing  a  major  strain  on  the  organization.  Engineers  had  to 
spend  significant  work  time  on  eBay  buying  parts.  Radar  testing  costs  escalated  significantly 
because  of  the  new  steps  required: 


1  Copyright  2015  Carnegie  Mellon  University.  No  warranty.  This  Carnegie  Mellon  University  and 
Software  Engineering  Institute  material  is  furnished  on  an  “as-is”  basis.  Carnegie  Mellon  University 
makes  no  warranties  of  any  kind,  either  expressed  or  implied,  as  to  any  matter  including,  but  not 
limited  to,  warranty  of  fitness  for  purpose  or  merchantability,  exclusivity,  or  results  obtained  from  use 
of  the  material.  Carnegie  Mellon  University  does  not  make  any  warranty  of  any  kind  with  respect  to 
freedom  from  patent,  trademark,  or  copyright  infringement.  This  material  has  been  approved  for 
public  release  and  unlimited  distribution.  DM-0002313. 
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1 .  Remove  an  airplane  from  operations. 

2.  Disassemble  the  radar  to  put  it  in  the  lab. 

3.  Calibrate  the  radar  in  the  lab  setting. 

4.  Complete  testing. 

5.  Reassemble  the  aircraft  with  the  radar. 

6.  Recalibrate  the  radar  to  the  airplane. 

Repercussions  of  this  problem  will  continue  even  after  the  equipment  becomes 
available  and  a  new  source  of  parts  is  established.  Both  physical  plant  and  process  changes 
will  be  required  to  reestablish  the  level  of  productivity  the  organization  had  when  its 
infrastructure  matched  the  technology  requirements  of  the  aircraft  systems  being  sustained. 

This  type  of  situation  occurs  with  some  frequency  and  can  be  summarized  as 

follows: 


•  An  organic  sustainment  organization  with  valuable  facilities  and  skills  is 
already  in  place. 

•  A  new  technology  makes  parts  obsolete  and  leads  to  new  engineering  design 
work. 

•  The  sustainment  organization  needs  to  upgrade  both  skills  and  facilities  to 
meet  the  demand. 

•  While  waiting  for  these  changes  to  be  completed,  the  sustainer’s  efficiency  is 
compromised,  and  “mission  capable  availability”  is  diminished. 

•  Until  process  and  tools  stabilize,  quality  often  suffers. 

Part  of  the  problem  for  the  sustainers  is  the  familiar  “color  of  money”  problem;  by 
law,  specific  funding  sources  must  be  applied  to  specific  uses.  Funding  for  product 
modernization  is  supplied  by  the  acquisition  budget  and  arrives  via  the  program  office. 
Funding  for  developing  organic  sustainment  capability  typically  comes  from  the  life-cycle 
command  function.  Delays  in  funding  to  update  facilities  and  processes  will  eventually  cause 
problems  in  mission  performance.  Detecting  this  situation  is  nearly  impossible  using 
spreadsheet  analysis  alone. 

At  least  five  distinct  stakeholders  are  present  in  sustainment  work: 

1 .  Actual  operators,  who  represent  the  mission-use  viewpoint 

2.  Strategic  planners,  who  review  threats  to  the  existing  system  and 
opportunities  for  new  system  capability  based  on  changes  in  technology 

3.  Sustaining  engineers,  who  must  address  requests  for  new  capabilities  as  well 
as  addressing  the  effects  of  external  changes  to  existing  subsystems  (e.g., 
software  changes  to  sensors  or  communications.) 

4.  The  sustainers’  management  team,  which  must  invest  in  facilities  and 
organizational  capabilities  and  retain  talent  with  product  domain  knowledge 

5.  The  program  office,  which  is  responsible  for  the  flow  of  funds  and  for 
promoting  the  program  to  all  stakeholders 

All  of  the  stakeholders  have  their  own  definitions  of  value  or  utility.  Each  also  has  a 
different  timeframe  for  decisions;  hence,  each  stakeholder  may  perceive  any  of  the  others 
as  delaying  the  response.  The  dynamic  is  one  of  constant  change  since  technology  may 
change  and  new  threats  arise  at  any  time.  Figure  1  shows  the  interaction  of  the 
stakeholders  and  the  potential  for  various  gaps  in  the  desired  performance. 
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Figure  1.  Stakeholders  and  Gaps 

Rationale  for  a  Simulation 

Sustainment  is  part  of  a  dynamic  and  changing  system.  Different  forces  are  stronger 
at  different  times,  and  there  are  time  delays  between  decisions  and  outcomes  that  hide 
valuable  information  from  decision-makers.  Any  thought  of  perfect  decisions  point  toward  an 
unattainable  ideal.  The  purpose  of  the  simulation  model  is  to  increase  the  likelihood  of  better 
decisions  by  forecasting  several  potential  futures  and  examining  the  consequences.  Even  if 
the  decision  is  not  optimal  at  the  start,  the  simulation  indicates  ways  to  make  course 
corrections  in  a  more  timely  fashion. 

Simulations  use  scenarios  of  input  and  change  to  make  these  forecasts.  Each 
scenario  assumes  a  possible  future  budget  allocation  and  a  future  set  of  demands  on  both 
the  mission  and  the  sustainer.  The  simulation  then  provides  a  graphical  picture  of  future 
performance  and  potential  gaps  in  that  performance.  In  many  cases,  the  simulation  will 
show  how  a  decision  that  appears  sensible  at  first  actually  generates  a  significant 
performance  gap  within  just  a  few  months.  Since  some  decision  cycles  are  often  two  years 
or  more,  the  gap  may  be  one  that  cannot  be  overcome  in  a  shorter  timeframe  than  the  next 
decision  cycle.  This  information  may  be  useful  in  persuading  stakeholders  (funders)  to  make 
different  decisions  about  funding. 

The  Simulation  Tool  and  Model 

The  model  uses  the  systems  dynamics  method  described  in  Peter  Senge’s  (2006) 
book  The  Fifth  Discipline.  The  SEI’s  simulation  was  developed  using  Vensim  and  systems 
dynamics  methods  (Ventana  Systems,  n.d.).  This  type  of  simulation  uses  a  “stock  and  flow” 
model  common  to  systems  dynamics  (Figure  2). 
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Figure  2.  Stock  and  Flow  Model 

An  epidemic  is  a  good  example  because  the  nonlinear  behavior  and  external 
interactions  can  create  a  future  dangerous  situation.  The  boxes  indicate  “stocks”  that  can  be 
supplied  or  drained  by  “flows,”  represented  by  double  arrows.  Stocks  may  also  be  initialized. 
The  double  triangle  is  a  valve  that  controls  the  rate  of  the  flow.  The  source  of  a  single  arrow 
is  a  measure.  The  target  is  a  calculation  or  control  of  a  valve.  If  the  source  of  the  arrow 
shows  an  increasing/decreasing  value,  the  sign  +/-  implies  that  the  target  of  the  arrow  has 
the  same/opposite  impact  as  the  measure  of  the  source.  In  the  diagram,  an  increase  in  the 
susceptible  population  creates  an  increasing  number  of  contacts.  Similarly,  if  the  duration  of 
the  infection  grows,  the  recovery  rate  decreases  (the 

The  full  simulation  for  sustainment  is  quite  large  and  can  be  seen  in  the  SEI 
publication,  A  Dynamic  Model  of  Sustainment  Investment  (Sheard  et  al.,  2015).  A  smaller 
piece  showing  how  technology  demands  interact  with  mission  performance  appears  in 
Figure  3. 


MSi 


-44- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


Legend  and  Interpretation 
In  Vensim  the  +/-  signs  are  replaced  by  "S"=+  and 
" 0 "=-. 

The  red  colored  arrows  simply  represent  a  reinforcing 
loop  called  the  "Bandwagon  Effect".  This  effect 
simply  means  that  a  series  of  successful  missions  will 
increase  the  demand  for  future  deployments  and  an 
expansion  of  mission  goals.  The  demand  then 
generates  requests  for  new  capabilities.  In  turn  the 
additional  capabilities  result  in  greater  expectations 
of  mission  performance. 


Figure  3.  Mission  and  Strategy  Portion  of  the  Simulation  Model 

The  Vensim  user  interface  (presented  in  Figure  4)  provides  a  customizable  control 
panel  with  charts  and  slider  bars.  The  charts  show  the  performance  graph  of  individual 
variables.  The  slider  bars  are  a  simple  control  panel  for  testing  different  inputs  or  input 
equations  that  drive  the  simulation.  The  charts  can  be  selected  from  any  number  of 
available  variables  in  the  model.  Additional  scenarios  can  be  developed  to  show  how 
actions  taken  later  affect  results. 

This  interface  allows  decision-makers  to  use  slider  bars  to  set  different  funding 
allocation  strategies  and  changes  in  demand  and  immediately  see  the  effects  of  different 
actions. 
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Figure  4.  Vensim  User  Control  Interface 
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Calibration 

Calibrating  a  simulation  is  always  challenging.  A  model  must  be  simpler  than  a  real- 
world  situation  since  modeling  complete  reality  is  far  too  expensive  and  time  consuming. 
Each  simplification  involves  some  abstraction  from  reality,  resulting  in  some  redefinition  of 
data.  Sometimes  an  approximation  or  proxy  must  be  used  if  real  data  is  not  available. 

The  most  common  way  to  calibrate  the  model  is  to  begin  by  establishing  an 
equilibrium:  When  the  equilibrium  is  established,  each  stock  and  flow  appears  constant.  This 
approach  works  because  equilibrium  can  usually  be  established  by  manipulating  a  smaller 
subset  of  the  variables  and  formulas. 

A  total  of  26  flows  and  19  stocks  are  defined  in  the  SEI  model.  Actual  calibration  was 
performed  with  approximately  half  the  total  number  of  flows  and  just  five  of  the  stocks.  The 
equilibrium  values  have  to  be  reconciled  with  data  observations  of  the  real  system.  Two 
particular  assumptions  show  the  complexity  of  the  abstraction: 

•  Enhancement  Requests:  Every  enhancement  request  was  counted  as 
having  the  same  size  and  effect  on  the  developers.  This  is  clearly  inadequate 
for  the  longer  term  viability  of  the  model.  Requests  come  in  different  sizes — 
some  are  big  and  complicated,  and  some  are  much  smaller  and  easier.  For 
improved  accuracy,  the  requests  are  considered  as  a  set  of  sizes  from  very 
small  (VS)  to  very  large  (VL). 

•  Staff  Capabilities:  It  was  also  necessary  to  connect  process  capability  and 
organizational  capacity.  We  chose  the  simple  formula: 

Sustainment  capacity  =  A*  (number  of  capabilities)  *  (number  of  staff), 
where  A  is  some  numeric  value  that  helps  to  achieve  equilibrium  when 
calibrating.  This  formulation  suggests  that  staff  capabilities  are  closely  related 
to  process  capabilities,  which  has  been  observed  in  many  studies. 

Proceeding  this  way,  we  determined  that  a  capability  change  could  be  based 
on  training  days,  since  process  changes  had  to  be  supported  by  training 
days.  The  stable  solution  at  this  time  is  about  45  days  per  staff,  or  about 
9,000  total  capacity  across  an  organization  of  200  people.  Capacity  is 
diminished  by  staff  members  leaving  or  reductions  in  total  staff.  Improving 
capacity  requires  both  staff  and  training  employed  together. 

It  is  possible  that  neither  of  these  assumptions  will  be  valid  after  further  study,  but  the 
simulation  behavior  appears  acceptable  in  current  use. 

Developing  Scenarios 

Typically,  the  simulation  should  imitate  real  performance  and  should  break  where  the 
system  would  break.  Testing  the  simulation  must  include  running  scenarios  of  change  and 
correlating  the  changed  behavior  of  the  model  to  the  organization’s  behavior.  A  scenario  can 
be  described  using  the  elements  in  Table  1. 
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Table  1.  Scenario  Example 


ELEMENT 

EXAMPLE 

Scenario  name  and  short 
description 

New  mission  threat 

Context  of  operation 

Normal  sustainment  activity  with  a  steady  rate  of 
enhancement  requests 

Stimulus 

Aircraft  squadron  runs  a  normal  mission  to  destroy 
anti-aircraft  capability.  However,  opponent  develops 
new  (technology)  capability  to  detect  aircraft. 

Response  (or  decision) 

Mission  planners  request  new  technology  for  aircraft 
mission  package  to  reduce  ability  to  detect  airplane. 

This  additional  work  places  a  strain  on  the  sustainment 
organization’s  ability  to  deliver  on  time.  Overtime  is 
requested  at  20%  rate  to  cover  additional  demands. 

Outcome  (what  the  simulation  tells 
us) 

Productivity  declines  over  time  anyway. 

Additional  scenarios  can  be  tested  simply  by  changing  any  of  the  first  three 
elements.  For  example,  adding  test  equipment  represents  a  potential  response.  Operating 
during  wartime  is  a  new  environment.  Finally,  an  example  of  a  new  stimulus  would  be  an 
enemy’s  new  capability  to  detect  a  stealth  aircraft.  Similar  scenarios  may  be  generated  by 
any  input,  whether  from  customer  demand,  operational  performance,  or  sustainment 
performance.  Each  new  scenario  may  have  different  effects  on  the  various  outcomes  in 
terms  of  productivity,  customer  satisfaction,  and  the  ability  to  perform  missions. 

Results 

Four  distinct  scenarios  were  tested  during  the  research.  The  results  of  the 
“Sequestration”  scenario  are  described  in  this  section. 

In  sequestration,  the  stimulus  was  a  20%  cut  in  funding.  The  initial  response  was  to 
cut  100%  of  the  training  budget.  The  outcome  for  that  particular  response  was  a  steady 
decline  in  productivity  for  as  long  as  sequestration  lasted.  An  alternate  response  was  also 
considered:  maintain  training  and  increase  investment  in  process  improvement.  The 
alternate  response  demonstrated  a  short-term,  six-month  loss  in  productivity  but  showed 
higher  levels  of  productivity  within  12-18  months.  Obviously,  there  are  practical  limits  to 
improving  productivity,  but  the  model  assumes  that  the  capital  investment  is  wisely  spent. 

Figure  4  also  illustrates  the  possibility  of  considering  alternatives  based  on  changing 
assumptions  about  either  the  baseline  or  the  stimulus  event  (represented  by  the  box  labeled 
“Alternate  Baselines”). 

The  other  three  scenarios  we  investigated,  which  are  described  in  the  larger  paper, 

were 


•  Gating  the  demand:  The  sustainment  organization  chooses  to  deny  a  number 
of  requests  because  it  does  not  have  the  capacity  to  fulfil  them  all.  What  will  it 
do  to  sustainment  performance  and  to  the  mission  performance  assessment? 

•  New  threat,  no  budget:  An  opponent  develops  a  technology  that  could 
compromise  mission  performance.  The  system  is  upgraded,  but  there  may 
not  be  money  for  training  and  tooling.  How  long  does  it  takes  for  mission 
performance  to  decline  and  by  how  much? 


MSi 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-47- 


•  Underfunding  sustainment  investment:  Underfunding  occurs  when  the 
sustainers  are  made  responsible  for  the  support  of  a  new  technology,  but 
there  is  no  budgetary  provision  to  support  tooling,  process  changes,  and  staff 
training.  Will  this  affect  mission  readiness,  and  how  long  will  it  be  before  the 
results  are  observed? 

These  initial  scenarios  are  not  completely  independent.  “New  Threat”  proved  to  be  very 
similar  to  “Underfunding.”  The  advantage  of  documenting  the  scenario  carefully  is  the  ease 
with  which  different  scenarios  can  be  tested. 

Outcomes  and  Future  Work 

The  Process  Resource  Team  and  the  Naval  Air  Weapons  Lab  worked  with  the  SEI 
on  this  model.  Their  confidence  in  the  model  was  sufficient  enough  for  them  to  purchase 
Vensim  and  to  seek  a  longer  term  relationship  with  the  SEI  to  study  and  improve  the  model. 

Several  kinds  of  improvement  to  model  are  possible.  The  most  obvious  is  to  improve 
calibration,  which  will  require  some  additional  data.  In  particular,  concepts  associated  with 
mission  capability,  mission  performance,  and  demands  for  new  capability  need  more  precise 
definitions  and  better  measurement  data.  Another  abstraction  that  should  be  reconsidered 
would  address  potential  trades  between  process  capability  and  individual  capability.  The 
initial  model  assumes  these  are  the  same  measure. 

The  Naval  Air  Weapons  Lab  plans  to  extend  the  use  of  the  model  to  other  programs 
within  NAVAIR.  The  future  practicality  of  the  simulation  model  depends  on  both  our  ability  to 
calibrate  the  model  to  the  system  of  interest  and  whether  the  scenarios  tested  represent 
actual  experience.  In  any  case,  the  simulation  will  almost  certainly  help  the  organization 
develop  better  long-range  plans  and  improve  cost  and  schedule  risk  mitigation. 
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